Now we've migrated Fenix and related repositories into m-c. It would be really nice to run mochitests (and other automated tests) on Fenix, not on TestRunner since TestRunner doesn't support dynamic toolbar, pull-to-refresh, etc.

Hey folks, can we start a conversation about this here? I think some of us talked about this before, but never in one general conversation. I know that running tests on the code we actually ship is a good idea, but we use test runner app for a reason - I think for performance reasons. So, do we want to run all the tests on Fenix or only some critical ones? If running all the tests on Fenix would take too long, is there anything we could do about it? Share your thoughts and ideas!

Also, let me know if there is anyone else I need to add to the conversation.

(In reply to [:owlish] 🦉 PST from comment #1)

Hey folks, can we start a conversation about this here? I think some of us talked about this before, but never in one general conversation. I know that running tests on the code we actually ship is a good idea, but we use test runner app for a reason - I think for performance reasons.

If performance is the concern, do we have any concrete data showing that using Fenix is significantly slower or consumes more resources compared to the test runner package? If not, and we need to gather this data, we could likely accomplish this by running the wdspec tests via geckodriver. In my opinion, geckodriver is currently the only tool capable of running tests with a different package, making it suitable for this task.

So, do we want to run all the tests on Fenix or only some critical ones? If running all the tests on Fenix would take too long, is there anything we could do about it? Share your thoughts and ideas!
GPT

I can only speak to the wpt tests, specifically wdspec. For these tests, it is essential that they run with a package actually used by our consumers. No one uses the test runner package, and we've encountered numerous missing features in it that prevent us from testing the features we need to support. But that's bug 1880800.

Also, let me know if there is anyone else I need to add to the conversation.

I would suggest that we get someone from the perf-test team added. Maybe Greg has some more details to share.

I don't have a super strong opinion on this other than to point out that the test runner gets built at a much earlier point of the build process than the Fenix app, so switching test suites over to it will incur a non-negligible delay in seeing results in CI (which in turn will make life more difficult for our code sheriffs). We should be thoughtful about which test suites are most likely to be impacted by the Fenix/AC layers on top of GV so we're only taking that delay where truly necessary.

haven't we been shipping fenix successfully for years without mochitests? Any regressions that are fenix specific, where are we doing the fixes and writing tests to catch those cases?

That's because, for example, for pull-to-refresh and dynamic toolbar related bugs, we have been paying extra costs to write JUnits tests and maintain the JUnit tests. For example there were a race condition in such JUnit tests, we had to address the race in bug 1660357, actually the race was addressed in mochitests long time ago.

so presumably we would rewrite junit tests in mochitests if we had the ability to run them on fenix? I don't understand the test runner side of things, I assume if we can run perf tests (browsertime) on fenix we could run mochitests on fenix.

Yeah, once after we started running mochitests on Fenix, we'd write any new tests as mochitests rather than JUnit and port the existing test cases to mochitest.

Hiro, is this something you still think we should move forward with? If so I can help bring in the necessary people. I just see that there was back and forth, and no updates in the last 3 months.

I would say yes.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #3)

I don't have a super strong opinion on this other than to point out that the test runner gets built at a much earlier point of the build process than the Fenix app, so switching test suites over to it will incur a non-negligible delay in seeing results in CI (which in turn will make life more difficult for our code sheriffs). We should be thoughtful about which test suites are most likely to be impacted by the Fenix/AC layers on top of GV so we're only taking that delay where truly necessary.

I have an idea to do kinda workaround for this concern. If we created a new job named mochitest-fenix or some such which runs a specific directly, not all other mochitests, it would be sufficient for us. Given that our test automation machinery has been learning, it will properly learn the relationship between our implementation and the specific mochitests at some point, then the specific mochitest will be run by ./mach auto or something like that.

Coming from https://bugzilla.mozilla.org/show_bug.cgi?id=1943329#c8

Probably there are a lot of things to consider and I see a nice discussion already.
For my two cents (an optimistic view) after a quick look I think in Fenix based on the scenarios tested I'd say we could have:

• JUnit tests for quick unit tests
• Espresso tests for general functionality integration tests
• Mochitests for browser integration tests

The integration tests could also be augmented with screenshot testing - would be quick and simple to have such a test for fullscreen toggling.

These should be separate tasks, not needing to run everything for simple changes.
We should also look into ensuring that ./mach try --preset auto is reliable in knowing which tests (of each type and from Fenix, AC, GV) to run for each change. Which based on Ryan's comment will mean a bit bigger tests infra changes.

I took a quick poke at this while getting geckoview_example working in mozperftest for mochitest. Geckoview_example worked with only a small change to the hardcoded activity name in RemoteProcessMonitor.launch. For fenix though, the browser is unable to connect to the http://mochi.test:8888 URL for some reason - not sure what could be preventing that connection.

If we get that connection working, I think the next issue to resolve might be writing to the mochitest.log file on the device (where the test results are pulled from to the host) but that's just a guess.

I managed to get Fenix to connect to the mochitest server at http://mochi.test:8888 but I had to change mochi.test to the IP of the host machine.

At that point, the page did load, but no tests were running. I looked into what was happening there a bit, and it seems like the issue might be related to not having access to SpecialPowers.Services. The window has SpecialPowers, but not the Services variable which is used for a number of things in mochitest. The failure is happening in testing/mochitest/chrome-harness.js afaict (can't get debug info from it) when we're trying to run the readConfig method, and that ends up setting the params to undefined resulting in failures during the button hook up method hookUp() in testing/mochitest/tests/SimpleTest/setup.js.

If anyone knows how to get the SpecialPowers.Services working, I'd be curious to know to see if we could get any further on running fenix in mochitest.

I suspect the custom test profile isn't getting loaded. mochi.test is defined in proxyAutoConfig as part of the prefs we pass into the custom test profile. This is generated and then copied to the emulator/device and is supposed to be in the proper location and referenced upon launch. Part of the profile is the specialpowers extension.

If the profile is there and recognized (lets say another reason mochi.test isn't working), then we can assume specialpowers is there and available, there could be issues in the bootstrapping, probably a JS error seen in logcat.

Thanks for the info :jmaher! That was helpful.

I can confirm that I'm now able to connect to mochi.test in the browser so I believe that the profile is getting loaded. However, I'm having the same issue as before where the mochi.test page loads, but no tests run.

From about:debugging, when I inspect the browser on the device, I found that the specialpowers, and the mochikit web extensions are not getting loaded - all the addons/webextensions I see there are internal ones. I also see a SpecialPowers is not defined error in the console for the mochi.test page.

So I verified that the permissions/ownership of the test_root/profile folders are good.

I managed to find an additional error in the adb logcat logs which suggests a component is unavailable - the ExtensionContent.sys.mjs file. Not sure what is using that or trying to use it. Does anyone know if this file/component is available in Fenix?

Here's the full error log that I found which occurs right at the same time as we load the mochi.test page:

04-11 12:08:17.165 23600 23655 E Web Content: [JavaScript Error: "[Exception... "Component is not available"  nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)"  location: "JS frame :: resource://gre/modules/ExtensionContent.sys.mjs :: ScriptCache/< :: line 171"  data: no]"]
04-11 12:08:17.165 23600 23655 E Web Content: [JavaScript Error: "[Exception... "Component is not available"  nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)"  location: "JS frame :: resource://gre/modules/ExtensionContent.sys.mjs :: ScriptCache/< :: line 171"  data: no]"]
04-11 12:08:17.166 23600 23655 E Web Content: [JavaScript Error: "TypeError: can't access property "manifestFile", params is undefined" {file: "http://mochi.test:8888/tests/SimpleTest/setup.js" line: 353}]
04-11 12:08:17.166 23600 23655 E Web Content: hookup@http://mochi.test:8888/tests/SimpleTest/setup.js:353:7
04-11 12:08:17.166 23600 23655 E Web Content: EventHandlerNonNull*@http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&logFile=%2Fdata%2Flocal%2Ftmp%2Ftest_root%2Flogs%2Fmochitest.log&fileLevel=INFO&consoleLevel=INFO&hideResultsTable=1&manifestFile=tests.json&dumpOutputDirectory=%2Fdata%2Flocal%2Ftmp%2Ftest_root&ignorePrefsFile=ignorePrefs.json:12:32
04-11 12:08:17.166 23600 23655 D GeckoViewContent[C]: handleEvent: pageshow
04-11 12:08:17.167 23600 23655 D GeckoViewAutoFill[C]: handleEvent: pageshow
04-11 12:08:17.168 23600 23655 D GeckoViewProgressDelegate[C]: handleEvent: pageshow
04-11 12:08:17.168 23454 23510 D GeckoViewProgress: receiveMessage: pageshow
04-11 12:08:17.169 23454 23510 D GeckoViewProgress: ProgressTracker handleEvent: pageshow
04-11 12:08:17.169 23454 23510 D GeckoViewProgress: ProgressTracker updateProgress
04-11 12:08:17.169 23454 23510 D GeckoViewProgress: ProgressTracker updateProgress data={"prev":55,"uri":"http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&logFile=%2Fdata%2Flocal%2Ftmp%2Ftest_root%2Flogs%2Fmochitest.log&fileLevel=INFO&consoleLevel=INFO&hideResultsTable=1&manifestFile=tests.json&dumpOutputDirectory=%2Fdata%2Flocal%2Ftmp%2Ftest_root&ignorePrefsFile=ignorePrefs.json","locationChange":true,"pageStart":true,"pageStop":false,"firstPaint":false,"pageShow":true,"parsed":true} progress=100
04-11 12:08:17.170 23600 23655 D GeckoViewContentDelegateChild[C]: handleEvent: MozFirstContentfulPaint
04-11 12:08:17.170 23454 23510 D GeckoViewProgress: ProgressTracker onStateChange: isTopLevel=true, flags=0xc0010, status=NS_OK
04-11 12:08:17.170 23454 23510 D GeckoViewProgress: ProgressTracker onStateChange: uri=http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&logFile=%2Fdata%2Flocal%2Ftmp%2Ftest_root%2Flogs%2Fmochitest.log&fileLevel=INFO&consoleLevel=INFO&hideResultsTable=1&manifestFile=tests.json&dumpOutputDirectory=%2Fdata%2Flocal%2Ftmp%2Ftest_root&ignorePrefsFile=ignorePrefs.json
04-11 12:08:17.170 23454 23510 D GeckoViewProgress: ProgressTracker stop
04-11 12:08:17.170 23454 23510 D GeckoViewProgress: ProgressTracker updateProgress
04-11 12:08:17.171 23454 23510 D GeckoViewProgress: StateTracker onStateChange: isTopLevel=true, flags=0xc0010, status=NS_OK loadType=1
04-11 12:08:17.171 23454 23454 I GeckoSession: handleMessage GeckoView:PageStop uri=null
04-11 12:08:17.173 23454 23510 D GeckoViewContentDelegateParent: receiveMessage: DispatcherMessage
04-11 12:08:17.174 23454 23510 D GeckoViewModule: dispatch GeckoView:IsPdfJs, data=null
04-11 12:08:17.174 23454 23510 D GeckoViewContent: onEvent: event=GeckoView:IsPdfJs, data=null
04-11 12:08:17.174 23454 23526 I SessionStorage/AutoSave: Save: Load finished
04-11 12:08:17.174 23454 23510 D GeckoViewModule: dispatch GeckoView:IsPdfJs, data=null
04-11 12:08:17.174 23454 23510 D GeckoViewContent: onEvent: event=GeckoView:IsPdfJs, data=null
04-11 12:08:17.175 23600 23655 D GeckoViewContentDelegateChild[C]: handleEvent: MozFirstContentfulPaint
04-11 12:08:17.175 23600 23655 E Web Content: [JavaScript Error: "NS_ERROR_NOT_AVAILABLE" {file: "resource://gre/modules/ExtensionContent.sys.mjs" line: 171}]
04-11 12:08:17.175 23600 23655 E Web Content: [JavaScript Error: "NS_ERROR_NOT_AVAILABLE" {file: "resource://gre/modules/ExtensionContent.sys.mjs" line: 171}]